Does tagging improve the navigation of online recorded lectures by students?

Students more and more have access to online recordings of the lectures they attend at universities. The volume and length of these recorded lectures however make them difficult to navigate. Research shows that students primarily watch the recorded lectures while preparing for their exams. They do watch the full recorded lectures, but review only the parts that are relevant to them. While doing so, they often lack the required mechanisms to locate efficiently those parts of the recorded lecture that they want to view. In this paper, we describe an experiment where expert tagging is used as a means to facilitate the students’ search. In the experiment, 255 students had the option to use tags to navigate 18 recorded lectures. We used the data tracked by the lecture capture system to analyze the use of the tags by the students. We compared these data to students who did not use the tagging interface (TI). Results show that the use of the TI increases in time. Students use the TI more actively over time while reducing the amount of video that they view. The experiment also shows that students who use the TI score higher grades when compared with students who use the regular interface. Introduction During their academic careers, students spent a great number of hours attending lectures. More and more universities record these lectures and provide them online to their students (Leoni & Lichti, 2009). Traditionally, metadata is created by dedicated professionals (Gruber, 1993); tags, on the other hand, are a form of user-created metadata (Mathes, 2004). Tags are textual keywords and phrases that can link to a number of different resources (O’Reilly, 2005). In this case, they are linking to locations within the recorded lectures. We believe that expert tagging has the potential to be a good fit with recorded lectures because most lecture capturing systems (LCS) currently lack sufficient support for students to navigate the recorded lectures efficiently and because tags have a low barrier to entry and relatively low cognitive costs for the expert (Mathes, 2004).The goal of this study is to investigate the use of tags as a navigational aid for students. In this study, we provided all students with the same set of tags constructed by an expert using a tagging protocol. We will address the following research questions: 1. When do students use the tags? 2. Does the availability of tags increase the navigation speed of students? Do the tags help them to more quickly locate the parts of the recorded lectures they want to view? 3. Do students who use the tags score better on the exam? British Journal of Educational Technology (2013) doi:10.1111/bjet.12121 © 2013 British Educational Research Association We will first describe recorded lectures and tags in more detail; then, we outline the method used, the results from the study, and finally, we will discuss the conclusions. Recorded lectures Survey-based research conducted both in the USA and Australia with a combined total of more than 8000 respondents showed that most students express a preference for courses accompanied by online recordings of the lectures (Gosper et al, 2008; Traphagan, 2006; Traphagan, Kucsera & Kishi, 2010; Veeramani & Bradly, 2008). These recorded lectures were created using an LCS. An LCS consists of a dedicated capture appliance in the lecture hall, capturing audio, video and the VGA signal of the lecturer’s laptop or desktop. The capture appliance can be started, stopped and monitored remotely by a system administrator. The capture appliance automatically synchronizes the recorded audio, the video and the VGA signal into a single interface stored on the server. The lecture can be viewed live or on demand by the user using a web browser. Because they are integral recordings from live lectures, the recorded lectures in this study are usually 40–45 minutes per recording. Research on 1200 students in the Netherlands, conducted by Gorissen, van Bruggen and Jochems (2012a, b) using surveys and interviews showed that students typically do not watch the full recorded lecture from start to finish. Instead, they jump to the parts of the lecture they want to (re)view and only watch those segments. Analysis of their navigation behavior showed that it often takes several jumps within the recording to find those parts they want to review. The research also showed that students use recorded lectures primarily while preparing for exams. Students use the recorded lectures as part of their regular learning activities. One other important activity is note taking. Taking notes during lectures is considered to be a key component of academic literacy (Badger, White, Sutherland & Haggis, 2001). Lecture notes provide an encoding function: they help students learn and remember the information from the lecture. They also provide a storage function, preserving the information provided during the lecture for later use (Anderson & Armbruster, 1986). Bligh (1998, pp. 129–147) lists an overwhelming amount of research that shows that note taking aids students in their learning process, and there is strong evidence that note taking leads to higher achievement (Kiewra, Practitioner Notes What is already known about this topic • Students prefer to use recorded lectures. • Students use recorded lectures mainly as part of their preparations for the exam. • Students use their notes to find the relevant parts of the recorded lectures. What this paper adds • Our research analyzes the use of tags, added by an expert, as a way for students to find the relevant parts of recorded lectures more easily. • The paper compares the use of the recorded lectures with those tags and without them. • The paper shows that students that use the tags score significantly higher for the exam. Implications for practice and/or policy • The paper shows that providing tags alongside recorded lectures can help students review recorded lectures more efficiently. 2 British Journal of Educational Technology © 2013 British Educational Research Association 1989). Students use their notes to find the relevant part of the recorded lecture that they want to (re)view. Most LCS, however, do not offer students easy textual clues. Students are limited to using a time-based video slider or a slide-based view to find those relevant parts. Tags Tags became popular as part of the Web 2.0 concept, allowing anyone to mark content with descriptive terms, in essence, adding their own explicit user-generated metadata (Mathes, 2004). We can identify a number of functions that tags perform (Golder & Huberman, 2006): • Identifying what (or who) the recording is about. This is the most common function of tags; • Identifying what it is. For example, a question or an example; • Self-reference. The tag identifies the tagger or begins with “my,” like “myquestion”; • Refining categories. These tags do not stand alone without contextual knowledge, for example, “Question 2”; • Identifying qualities or characteristics of the resource. For example, funny, stupid, difficult; and • Task organizing. These tags relate to performing a task, for example, “to read.” Popular sites like del.icio.us (Delicious, 2013) allowed, and still allow, users to add bookmarks to websites combined with tags. Those tags are not only visible to the users that added the bookmarks but also to other users of the site. Combined, they form a folksonomy of tags (Vander Wal, 2007), a flexible bottom-up, user-created categorical structure. Although tagging is quite common now and supported by many websites, these social tagging systems were a significant step up from the original idea of metadata and being created and maintained by professionals. Our research adds to the existing research by extending the use of tags to recorded lectures. Method The research questions were investigated at the Eindhoven University of Technology (TU/e) in the Netherlands. The university uses the Mediasite LCS by Sonicfoundry (Sonicfoundry). They have five capture appliances used to create recorded lectures. Currently, the TU/e Mediasite repository holds about 8000 recordings with a total length of over 5000 hours of video. All recordings are available online; students can view them in their browser, both at the university and from home. No downloadable versions of the recordings are provided. The recorded lectures created at the TU/e are typically about 40–45 minutes long, containing one full lecture each. The number of recordings per course depends on the number of lectures for that course; the average is 16 recordings for a single course, and the maximum number of recorded lectures for a single course is 54 recordings. During a previous stage of our research (Gorissen, van Bruggen & Jochems, 2012c, 2013), we had shown that surveys were not a reliable method to collect data on students’ use of recorded lectures. Instead, we choose to collect the data using logging of their actual use during a course which lectures are being recorded. Course C01 is a course at the faculty of Industrial Engineering & Innovation Sciences of the TU/e. Students who participate in the course come from a number of different departments within the university. Most of the students (66%) are from the Industrial Engineering department; another substantial group of students (23%) is from the Innovation Sciences department. The course consists of an introduction in empirical research. Students learn how to translate real-life questions into research questions, and they learn how to create and evaluate a research design. In the second part of the course, they get hands-on training using SPSS. This part of the course is taught in English and was selected for the tagging experiment because many students use the recorded lectures of this part of the course. Grading for the course is done in two parts, and grades are registered for both individual parts and for the complete course. The first partial grade (C01P01) consists of the combined result for a 90-minute written test and an assignment. The second partial grade (C01P02) consists of a 3-hour laptop test. Each Does tagging improve the navigation of recorded lectures? 3 © 2013 British Educational Research Association partial grade (C01P01 and C01P02) needs to be 5.0 or higher on a 10-point scale. If that is the case, the grade for C01 = .5 × C01P01 + .5 × C01P02. This combined grade needs to be 6.0 or higher on a 10-point scale. Figure 1 shows an example of the browser-based player students regularly use to view the recorded lectures. We will refer to this player as the regular interface (RI). We chose to implement the tagging interface (TI) as a separate layer on top of the RI. This eliminated the need to make changes to the existing environment, keeping the experiment restricted to just the course selected. Figure 2 shows an example of the TI. Within the TI, students have all the navigational options that the RI has, with the addition of the tags displayed on the left-hand side. We could not allocate students to TI or RI conditions. Students were free to select the RI or the TI whenever they retrieved a recorded lecture. When students wanted to view the recorded lectures, they had to log in. The system logged which students viewed the recorded lectures and in case of the TI, the system logged which tags the students used to navigate through the recorded lectures. To determine where to place the tags, we used signals, or signposts, provided by the structure of the lecture. Exley and Dennick (2004) describe a number of possible lecture structures a lecturer can choose from. Table 1 describes the structure types and suggests possible tag indicators per structure type. Exley and Dennick distinguish a number of different types of statements a lecturer can use to inform students about the lecture organization: signposts, frames, foci and links. Bligh (1998, p. 84) refers to these signals as macro signals. Signposts are statements that signal the direction the lecturer is going to take. An example of a signpost is as follows: In this first lecture I am going to talk to you about the structure of the course, the topics that are part of the exam, the structure of the exam, which is laptop based, and the best way to prepare for it. But first I am going to show you an example. Figure 1: Example of the regular interface (RI) 4 British Journal of Educational Technology © 2013 British Educational Research Association Figure 2: Example of the tagging interface (TI) Table 1: Lecture structures Description Possible tag indicators Classical Lecture is series of related entities, describing their features or properties. Start of new entity Sequential Lecturer goes through a simple sequence of related subtopics that underpin the main topic and form a logical and coherent “narrative” with a specific conclusion. Start of new subtopic Process Uses the sequence of components within a process (eg, in biochemistry, ecology, geology, economics) as the framework for the lecture Start of new process step Chronological Uses a temporal or historical sequence to structure the lecture Start of new time sequence Spatial Uses the spatial relationships between entities as a structure, for example, in anatomy and embryology, geography or architecture Start of new spatial relationship Comparative The lecturer sets up a debate between competing ideologies, concepts, methods, procedures or techniques. Start of new ideology Start of new concept Start of new method Start of new procedure Start of new technique Induction and deduction Induction is the process by which observations, facts and evidence are synthesized to form theories, rules and laws. The opposite process by which theories and rules are used to predict and calculate facts about the world is known as deduction. Start of new theory Start of new rule Start of new observation Both processes can be used to structure a lecture. Start of new fact/evidence Start of new deduction or induction step Problems and case studies Case studies can be used to structure lectures by bringing together conceptual understanding and reasoning with real-life, relevant situations. Start of case/problem explanation Does tagging improve the navigation of recorded lectures? 5 © 2013 British Educational Research Association Frames are statements that indicate the beginning and end of topics and sections. An example of a frame follows: OK, to start off with our first topic, let’s have a look at the way you can determine the size of a sample. Foci are statements that highlight and emphasize key ideas, definitions and concepts. An example of foci statements is as follows: Now during the exam, you will get data that has funny stuff in it. I won’t tell you that that is the case; I will expect you to know that is the case, so make sure you check for that first. Links are statements connecting to other sections of the lecture or prior knowledge and experience. Two examples of a link statement follow: As you should remember from the first part of the course . . . Contrary to what we did during the previous example, we now . . . The structure of the slides also provides a signal for the lecture organization. The slides contain titles, lists of important topics, schemas with the structure of the topics, etc. Based on these signals, we created the following tagging protocol for our experiment: 1. Examine the lecture structure (see Table 4). This gives an indication of the sort of possible tag indicators that signal useful tags. 2. Playback the recorded lecture, and while playing, listen to oral signals by the lecturer that indicate signposts, frames, foci or links. 3. Mark potential tags. Pause the recording, write down the time code along with potential tag title and a short description of the tag. 4. After completion of the tagging process, the tags, descriptions and time codes were added to the tagged player system. 5. Always add a tag at 00:00:00, indicating the beginning of the recording. This gives the student an easy way to return to the beginning of the recording. Using the tagging protocol, the recorded lectures were tagged by hand within 2–3 hours after each recording had been created. All tags have a title (always visible), a description (visible when the user hovers over the title) and a time code (not visible to the user). Once logged in, tags could be added and edited as shown in Figure 3. Tags were added to the recorded lectures by the main author of this article using a management interface constructed for the experiment. After a recorded lecture had been tagged, the professor for the course reviewed and approved the tags and descriptions. A total of 18 recorded lectures, each 40–45 minutes in length, were tagged, spanning the full 9 weeks of the second part of course C01. Table 2 shows, as an example, the tag titles, descriptions and time codes used for the C01 Lecture 11b recording. Usually, the lecture started with an introduction of the topic or a summary of the topic covered to that point. That introduction was then often followed by a demonstration of that topic using SPSS. For the C01 course, a total of 18 recorded lectures (C01P02L1 through C01P02L18) were created. Each recorded lecture is on average 40–45 minutes long. Half of all tags have been added to the first 14–16 minutes of the recordings, the rest to the remaining 25–30 minutes. The total number of tags added was 202, the average number of tags per recorded lecture was 11 (SD = 3.6). Given our three research questions, we performed a number of analyses: 1. When do students use the tags? a. When do the students use the TI? b. Does the use of the TI increase over time? 6 British Journal of Educational Technology © 2013 British Educational Research Association 2. Does the availability of tags increase the navigation speed of students? Do the tags help them to more quickly locate the parts of the recorded lectures they want to view? a. How much video do students view per recording session using the TI compared with the RI? b. Do the students click more or less during a recording sessions while using the TI? 3. Do students who use the tags score better for the exam? a. Are the exam scores for students that use the TI higher? Results First, we will analyze the first research question: when do the students use the tags? During the time period of the experiment, November 2011 through January 2012, a total of 255 students viewed one or more of the recorded lectures for the second part of the C01 course. Of those 255 students, 172 students only used the RI to view the recorded lectures, eight students only used the TI and 75 students used both the TI and the RI during the timeframe of the experiment. We analyzed the use of the recorded lectures on the level of individual “recording sessions.” A recording session is defined as an uninterrupted period of time during which a learner views one single recorded lecture (Advanced Distributed Learning, 2004). If a student views multiple Figure 3: Adding and editing tags Table 2: Tags for C01 lecture 11b Tag title Description Time code New variables based on existing ones How to do this 00:00:00 Interaction variables How to add and interpret them 00:06:00 A crude specification test Ramsey’s “omitted variable test” 00:12:22 Ensuring proper models Creating new y variables 00:23:40 Outliers What is it and how to take care them? 00:32:12 Weekly not-on-the-exam fact Each last 5 minutes of a lecture 00:40:00 Does tagging improve the navigation of recorded lectures? 7 © 2013 British Educational Research Association recorded lectures consecutively, they are all considered to occur in their own individual recording session. This way a recording session can be labeled as either a TI session or a RI session. Figure 4 shows the number of recording sessions for the C01 course for both TI and RI on a weekly basis. It is clearly visible that the number of RI recording sessions (3477 in total during the experiment) is much greater than the number of TI recording sessions (879 in total during the experiment). The graph shows a significant increase in the number of recording sessions for both the RI and TI in the last 4 weeks of the experiment, in the weeks leading up to the exam for C01. Figure 5 shows in more detail the peak in number of recording sessions for both the RI and TI during the last 7 days before the exam. These results confirm that students most frequently view the recorded lectures during the week leading up to the exam (Gorissen et al, 2012b). An analysis of recording sessions per day shows that peaks in session numbers per day occur on Wednesdays and Thursdays (the lecture is on Wednesday), whereas the troughs often occur on Saturdays. Figure 4: Number of recording sessions for tagging interface (TI) and regular interface (RI) Figure 5: Number of recording sessions for tagging interface (TI) and regular interface (RI) during the last 7 days before the exam 8 British Journal of Educational Technology © 2013 British Educational Research Association Figure 6 shows the percentage of the total number of recording sessions where the TI was used per week. The dashed line shows the trend line. Analysis of the percentage in SPSS revealed a Pearson product–moment correlation coefficient of r = .775, n = 13, p = .002. The use of TI when compared with RI increased over time. Our second research question relates to the navigation speed of students when they use the TI. Do the tags help them to more quickly locate the parts of the recorded lectures they want to view? Figure 7 shows that although the absolute number of RI recording sessions is higher than the number of TI recording sessions, the average amount of video received during a TI recording session is higher for most of the recorded lectures. The figure shows the average amount of video (in minutes) per recording session for each recorded lecture for the RI and TI. It shows that, with the exception of the first three recorded lectures, for other recorded lectures, the average amount of received video per recording session is higher for the TI than for the RI. We did a further analysis into the differences of navigation speed between the RIs and TIs by comparing the amount of use of the interfaces over time. Research by Gorissen et al (2012b) shows that students use recorded lectures more when preparing for the exams. For this course, the exam was on January, 23 2012. Figure 4 shows a significant increase of the number of recording sessions after January 1, 2012. Figure 6 shows an increase in the use of the TI over Figure 6: Proportion tagging interface (TI) of total number of recording sessions per week Figure 7: Average amount of video (in minutes) per recorded lecture per recording session Does tagging improve the navigation of recorded lectures? 9 © 2013 British Educational Research Association time. We have taken these two developments into account while analyzing the second research question. For the RI, we calculated the amount of video received per recording session of the period before and after January 1, 2012. Table 3 shows that the average amount of video received per recording session for RI decreased from 13.07 to 12.29 minutes per recording session, a decrease of 5.9%. The students viewed less video per recording session for the recording sessions while using RI in the weeks leading up to the exam. For the TI, we calculated both the amount of video received and the number of times a tag was clicked during a recording session, again comparing the period before and after January 1, 2012. Table 3 shows the results of that comparison. Table 4 shows the same information as Table 3, but for the TI. It includes additional columns for the data related to the total and average times the students clicked on a tag in the TI. The table shows that the average tag click count per TI recording session more than doubles from 8.80 to 18.34. The average amount of video per recording session for the TI decreases from 20.05 to 13.20 minutes (−34.2%). Students use the tags in the TI more actively during the last weeks before the exam, yet view less video per TI recording session than before January 1, 2012. Our final research question was whether students who use the tags score better for the exam than students who do not use the TI. Does an increased us of the TI lead to a better grade for the second part of the course (C01P2_grade)? Because we could not allocate students to TI or RI conditions, we analyzed the data using linear regression with CO1P2 as dependent variable. To control for existing differences in exam performance, the results of the first part of the exam (CO1P1_grade) were used as a covariate. The number of sessions using TI (count_TI) as well as RI (count_RI) was used as predictor variables. The analysis was performed using students from whom we had both CO1P and CO1P2 scores (n = 167). We checked for multivariate outliers using the Mahalanobis’ distance (using α = .001 Table 3: Video received for regular interface RS** (n) Video received*